1. Field of the Invention
The present invention generally relates to an image processing apparatus and an image processing method. More particularly, the present invention relates to an image processing apparatus and an image processing method for carrying out image processing accompanied by conversion between a non-interlaced image signal and an interlaced image signal.
2. Description of the Related Art
So-called conversion of ‘interlaced→non-interlaced→interlaced’ is carried out in association with static image processing and moving picture processing (generically referred to as ‘optional image processing’, hereinafter) in a video camera apparatus or a digital camera apparatus (which may be generically referred to as a ‘camera apparatus’ hereinafter).
The conversion of ‘interlaced→non-interlaced→interlaced’ means a process of converting an interlaced image signal (described later) into a non-interlaced image signal, and then, returning the non-interlaced image signal into the interlaced signal.
Concerning a process of the conversion of ‘interlaced→non-interlaced→interlaced’, it is desired to reduce the number of parts/components required, to reduced required power consumption, and, to make it possible to easily obtain from image information an interlaced image to be displayed on a monitor or to be provided externally, and further, to make it possible to easily adapt to a period or clock timing at which an image is obtained by a camera device, and so forth.
These problems may be solved by improvement in LSI, FPGA or such (generically referred to as a ‘integrated circuit’ hereinafter) of a camera apparatus, integrated circuits provided in the periphery, external memories, integrated circuits for processing an image signal obtained from a camera device, and a method of controlling the external memories, provided for image processing.
FIGS. 1A through 1C illustrate an interlaced image signal format and a non-interlaced image signal format.
FIG. 1A illustrates an image on a monitor. As shown, in an example shown in FIG. 1A, an image (for one frame) on the monitor includes a total of 480 scan lines including 240 odd scan lines ODD(1) through ODD(240), and 240 even scan lines EVEN(1) through EVEN(240).
FIG. 1B illustrates an interlaced image signal.
As shown, an interfaced image signal is such that, in the above-mentioned total of 480 scan lines, an image signal of odd scan lines ODD(1) through ODD(240), i.e., 1st, 3rd, 5th, . . . , 479th scan lines are transmitted in sequence in the stated order, and after that, an image signal of even scan lines EVEN(1) through EVEN(240), i.e., 2nd, 4th, 6th, . . . , 480th scan lines are transmitted in sequence in the stated order. Such an arrangement of scan lines may be referred to as an interlaced arrangement, hereinafter, and an image signal having an interlaced arrangement may be referred to as an interlaced image signal.
FIG. 1C illustrates a non-interlaced image signal.
As shown, a non-interlaced image signal is such that the above-mentioned total of 480 scan lines are transmitted in the order, i.e., the odd scan line ODD(1), the even scan line EVEN(1), the odd scan line ODD(2), the even scan line EVEN(2), . . . , the odd scan line ODD(240), the even scan line EVEN(240). That is, the 1st, 2nd, 3rd, 4th, . . . , 479th and 480th scan lines of the image signal are transmitted in sequence in the stated order. Such an arrangement of scan lines may be referred to as a non-interlaced arrangement, hereinafter, and an image signal having a non-interlaced arrangement may be referred to as a non-interlaced image signal.
In a camera apparatus of the related art, when optional image processing is carried out on a static image or a moving picture taken by means of a camera device, the processing is carried out in the above-mentioned non-interlaced image signal state in many cases. On the other hand, when an image signal is provided to a display unit (simply referred to as a monitor), it is necessary to provide the image signal in the above-mentioned interlaced arrangement.
Therefore, it is necessary to convert a non-interlaced signal having undergone optical image processing into an interlaced signal to be provided to a monitor. Such conversion is carried out with the use of a memory (SDRAM, SRAM, DRAM, QDR, QDRII, RAM included in an integrated circuit or such). That is, the non-interlaced image is written in the memory in sequence, scan line by scan line. Then, reading is carried out from the memory in such an order to obtain an interlaced arrangement.
FIG. 2 shows an internal block diagram of a video camera apparatus in the related art.
An interlaced image signal taken by a camera device 1 shown in FIG. 2 is converted into a non-interlaced image signal by means of a circuit 2 for converting an interlaced image signal into a non-interlaced image signal (simply referred to as an ‘interlaced to non-interlaced circuit’ hereinafter) with the use of a frame memory 3. The non-interlaced image signal thus obtained undergoes optional image processing (i.e., a filter function process, an electronic zooming process or such) by means of an image processing part 4. After that, the non-interlaced image signal is converted into an interlaced image signal by means of a circuit 5 for converting a non-interlaced image signal into an interlaced image signal (simply referred to as an ‘non-interlaced to interlaced circuit’ hereinafter) with the use of a frame memory 6. Alternatively, a circuit 7 for converting a non-interlaced image signal into a VGA image signal may be used to convert the non-interlaced image signal into a VGA image signal.
The interlaced signal thus obtained is then transmitted to a connector 9 for externally providing the image signal or to an analog monitor 10 for displaying the image signal. On the other hand, the VGA image signal may be then transmitted to a connector 12 for externally providing the image signal or to a VGA monitor 11 for displaying the image signal. Alternatively, the image signal may be transmitted to a recorder 8 such as a hard disk recorder.
FIG. 3 shows a circuit configuration example for enabling monitoring an image signal which is in the middle of optical image processing, in comparison to the above-mentioned configuration of the video camera apparatus.
In the configuration of FIG. 3, the image processing part 4 of FIG. 2 is divided into image processing parts 4A and 4B. Image signals before and after undergoing optional image processing carried out by each of the image processing parts 4A and 4B are converted into interlaced signals by means of non-interlaced to interlaced circuits 5C, 5B and 5A, with the use of frame memories 6C, 6B and 6A, respectively. An image signal after being thus converted is transmitted to a respective one of externally providing connectors 9C, 9B and 9A, or to a respective one of analog monitors 10C, 10B and 10A.
FIGS. 4A, 4B and 4C illustrate flows of processing carried out by the respective functional blocks 2, 3, 4, 5 and 6 shown in FIG. 2.
As mentioned above, a static image or a moving picture taken by the camera device 1 corresponds to an interlaced signal, while optional image processing in the image processing part 4 is carried out on a non-interlaced image signal. The frame memory 3 is used for converting the interlaced image signal into the non-interlaced image signal.
That is, the interlaced to non-interlaced circuit 2 writes an interlaced image signal in the frame memory 3 in an order of being transmitted. Then, after the image signal is written in the frame memory 3 for one frame, the interlaced to non-interlaced circuit 2 reads the written image signal in such an order to obtain a non-interlaced arrangement. The thus-obtained non-interlaced image signal then undergoes optical image processing carried out by the image processing part 4. After that, the non-interlaced image signal is returned into an interlaced image signal by means of the non-interlaced to interlaced circuit 5 with the use of the frame memory 6. Thus, a total of the two frame memories 3 and 6 are required before and after the optical image processing for the purpose of signal conversion between the interlaced image signal and the non-interlaced image signal.
When an allowable delay time from an input image signal to an output image signal is equal to or more than 1 frame for example, optional image processing of the image processing part 4 carried out on a non-interlaced image signal is carried out at the same rate. Such a process will be simply referred to as a ‘same rate process’. FIG. 4B shows a time chart of the same rate process. Below, with reference to figures, a flow of operation of an image processing apparatus carrying out optional image processing in a same rate process will be described.
In FIG. 4B, from the camera device 1 to a signal path P4, an odd image signal ODD(A) and an even image signal EVEN(A) for one frame are output in sequence. The odd image signal ODD(A) first transmitted is written in the frame memory 3 in sequence. After the completion thereof, the interlaced to non-interlaced circuit 2 reads the thus-written odd image signal ODD(A) from the frame memory 3, disposes the read odd image signal alternately with even scan lines of the even image signal EVEN(A) which is subsequently transmitted, scan line by scan line, in sequence, and outputs a non-interlaced image signal thus obtained to a signal path P2 (ODD(A)/EVEN(A)).
Thus, the interlaced image signal is converted into the non-interlaced image signal. The image processing part 4 carries out a same rate process on the thus-provided non-interlaced image signal to obtain a resulting image signal ODD(A′)/EVEN(A′).
A thus-obtained image signal (ODD(A′)/EVEN(A′)) having undergone optional image processing carried out by the image processing part 4 is output to a signal path P3, and then, is converted into an interlaced image signal by the non-interlaced to interlaced circuit 5.
Specifically, the non-interlaced image signal ODD(A′)/EVEN(A′) having undergone the optional image processing is written in the frame memory 6. After a first half of the non-interlaced image signal ODD(A′)/EVEN(A′) having undergone the optional image processing has been written in the frame memory 6, the non-interlaced to interlaced circuit 5 reads only odd scan lines of the odd image signal ODD(A′) from the frame memory 6 in sequence, for a first half of odd scan lines of an interlaced image signal. After that, the non-interlaced to interlaced circuit 5 obtains odd scan lines ODD(A′) of an interlaced image signal with the use of odd scan lines ODD(A′) included in a second half of the non-interlaced image signal (ODD(A′)/EVEN(A′)) having undergone the optional image processing and transmitted subsequently, for a second half of odd scan lines of the interlaced image signal. The thus-obtained odd scan lines of the interlaced image signal are output to a signal path P4.
After that, the non-interlaced to interlaced circuit 5 reads even scan lines of the non-interlaced image signal having undergone the optional image processing which have been already written in the frame memory 6 for one frame at the present time, thus obtains even scan lines of the interlaced image signal, and outputs the thus-obtained even scan lines of the interlaced image signal to the signal path P4.
On the other hand, when an allowable delay time is one field, that is, when a delay of a signal from the signal path P1 to the signal path P4 should fall with one field, i.e., ½ frame, optional image processing should be carried out at a double rate. Such a process carried out at a double rate will be simply referred to as a double rate process, hereinafter. FIG. 5B shows a time chart of a double rate process.
In this case, different from the above-mentioned case of FIG. 4B, after writing odd scan lines ODD(A) included in a provided interlaced image signal in the frame memory 3, the interlaced to non-interlaced circuit 2 reads the thus-written odd scan lines, and disposes the read odd scan lines alternately with even image signal EVEN(A) subsequently transmitted, scan line by scan line. Thus, the interlaced to non-interlaced circuit 2 obtains a double rate non-interlaced image signal, which is then output to the signal path P2.
In this case, the non-interlaced to interlaced circuit 5 writes even scan lines EVEN(A′) included in a non-interlaced image signal ODD(A′)/EVEN(A′) having undergone optional image processing of the image processing part 4, and provides odd scan lines ODD(A′) to the signal path P4. After that, the non-interlaced to interlaced circuit 5 reads the written even scan lines EVEN(A′) from the frame memory 6. Thus, an interlaced image signal ODD(A′)/EVEN(A′) is provided to the signal path P4 with a delay of one field.
Further, as in the circuit example of FIG. 3, in a case where an image signal which is in the middle of optical image processing is to be extracted, a memory 7B is separately required for converting a non-interlaced images signal into an interlaced image signal. By sharing for this purpose the memory 7A which is originally used after the optional image processing is carried out, it may be possible to omit the necessity of such an extra memory. However, in this method, it is not possible to output an image signal having undergone the optional image processing to the monitor when the image signal in the middle of the optional image processing is extracted. As another method, control of a memory may be made in a time division manner such that both image signals before and after undergoing the optional image processing are output to the monitors. However, even in this method, there is a limitation to output image signals to many monitors simultaneously.
Further, generally speaking, there are two different periods or clock timing to obtain an image by a camera device, i.e., a 30-Hz period and a 60-Hz period. As shown in FIG. 1A, one page or one frame of image includes odd scan lines and even scan lines. In this case, there are a 60-Hz method and a 30-Hz method as will be described below. That is, in the 60-Hz method, an image is obtained at 60 Hz, and, with this timing being kept, an image only including odd scan lines and an image only including even scan lines are extracted alternately. In the 30-Hz method, an image is obtained at 30 Hz, the thus-obtained image is decomposed into an odd scan line image and an even scan line image, and the thus-obtained images are output in sequence in a time-series manner at 60 Hz.
In the 60-Hz method, odd scan lines and even scan lines alternately extracted have different times at which the images have been taken by a camera device. Therefore, when a non-interlaced image signal is obtained from conversion with the use of the odd scan lines and even scan lines, the images having the different times of being taken by the camera device are combined in such a manner that corresponding scan lines are adjacently disposed alternately. FIGS. 6B through 6D show an outline of an image obtained in this case, and FIG. 6A shows a time chart illustrating a change in the image along a time axis.
When images having different times of being taken by a camera device are thus combined into one image as mentioned above in such a manner that corresponding scan lines are adjacently disposed alternately (in FIG. 6A, O1 and E1, O3 and E4), the states of images OE12, OE34 shown FIG. 6A occur.
Assuming a case where a moving picture of an object currently moving rightward in the FIG. 6A is taken by a camera device, an image O1 including only odd scan lines (FIG. 6B) taken by the camera device in early timing and an image E1 including only even scan lines (FIG. 6C) taken by the camera device in timing delayed by timing corresponding to 60 Hz, are compared. Then, As shown in FIGS. 6A and 6B, the object has shifted rightward in FIG. 6C in comparison to FIG. 6B. As a result, when both are combined and a non-interlaced image signal is thus obtained, a contour of the object is in a saw-tooth state OE12 or OE23 as shown in FIG. 6D.
When an electronic zooming process as optional image processing is carried out on the thus-obtained non-interlaced image signal, a saw-tooth noise may occur in a resulting moving picture due to the above-mentioned difference in the time at which the images have been taken by the camera device. Similarly, when a filter process is carried out, a calculation error may occur in an integrated value. Such noise may occur as a result of, as shown in a part defined by a broken line of FIG. 6D for example, a contour of the object, which should be originally linear, changing to have saw tooth as mentioned above, then the optional image processing being carried out thereon, and further, the image signal being then converted into an interlaced image signal.
Japanese Laid-Open Patent Applications Nos. 62-217287 and 6-261299 and Japanese Patent No. 2731639 disclose related arts.